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ABSTRACT 


The main objective of this project aims at delivering the need for one 
of many applications of the Internet of Things domain. It is a 
combination of various domains such as image processing and 
robotics for development in the field of the Internet Of Things (IoT) 
now also termed as Internet Of Everything (IoE) after seeing its 
involvement in almost everything happening in our day to day lives. 
With increased dependency on home automation and wireless 
controlled equipment and gadgets, this project also shares some 
scope in home automation. To improve the functionality of gadgets 
and to increase their efficiency we thought of creating a device to 
cater to these needs and explore alongside our own interest in the 
field of the Internet of Things. This paper presents the design, 
functioning, and successful testing of a rover controlled wirelessly 
with help of hand gestures. Gesture Recognition has played important 
role in the field of Human-Computer Interaction (HCI). Vision-based 
hand gesture recognition provides a great solution to various machine 
vision-based applications by providing an easy interaction channel. 
For various automated machine control applications, an effective 
real-time communication approach is required. In this paper, a vision- 
based hand gesture-based approach is presented for providing a real- 
time control of Arduino-based robot. Combining all of these concepts 
into a single device that can recognize hand gestures and sends data 
wirelessly to remote devices for surveillance and other purposes. 
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With an increase in digitalization and automation in 
many industries, various problems and jobs which 
were once considered laborious for humans are now 
becoming effortless. Robots’ functionality and 
accuracy have proven to be efficient, clean as well as 
productive in various industrial and security sectors. 
This along with the help of other domains such as 
image processing and wireless connectivity over the 
internet is now becoming a major part contributing to 
the betterment of people in the coming future. The 
main objective of this project aims at delivering the 
above saying. It is a combination of various domains 
such as computer vision, HCI, and robotics for 
development in the field of Internet of Things (IoT) 


now also termed Internet of Everything (IoE) after 
seeing its involvement in almost everything 
happening in our day to day life. With increased 
dependency on home automation and wireless 
controlled equipment and gadgets, this project also 
shares some scope in home automation. 


To improve the functionality of gadgets and to 
increase their efficiency we thought of creating a 
device to cater to the needs and explore alongside our 
interest in the field of the Internet of Things. This 
paper uncovers Human-computer interaction using 
Hand Gesture Recognition. It presents the design, 
functioning, and successful testing of a rover 
controlled wirelessly with help of hand gestures. With 
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wireless connectivity and Bluetooth support for easy 
data transmission, this rover can be monitored and 
controlled on multiple devices. The camera module 
capture and send real-time footage to users remotely 
here as an Ip address link for live tracking of the 
environment surrounding the rover. Adding motor 
control gave us maneuverability and Arduino 
functions as the brains of this rover. To instruct this 
rover we took the help of serial communication to 
coordinate tasks between Arduino and python 
programming language. 


Literature Survey: 

In past, a lot of research work has been done on 
recognizing hand gestures efficiently. Systems were 
developed with applications in robotic control, 
gaming applications, and many more. There has been 
much research in the field of hand gesture recognition 
for controlling robots. 


Rafiqul Zaman Khan et al [2] discussed the issues in 
hand gesture recognition and provided the literature 
review on different techniques used for Hand Gesture 
Recognition. 


R. Pradipa et al [4] performed an analysis of different 
Hand Gesture Recognition Algorithms and provided 
their advantages and disadvantages. 


A Computer Vision-based approach for gesture 
recognition includes four phases commonly Image 
Acquisition, Pre-processing, Feature Extraction, and 
Classification as shown in Fig. 1. Firstly, the image of 
the user providing gestures is collected through a 
webcam and is followed by some preprocessing. 
Then, the hand portion is extracted and classified into 
the category of a particular gesture [5]. 


F. Arce et al [6] illustrated the use of an 
accelerometer based on 3 axes to recognize Hand 
Gestures 


Proposed System: 

The system hardware includes Arduino Uno 
microcontroller, Dc Motors, L298N motor Driver, 
HC-05 Bluetooth Module, LED, and_ Esp3- 
Cam.Esp32 camera is a small size, low power 
consumption camera module based on ESP32. It is 
widely used in IoT applications such as wireless 
video monitoring, Wifi image upload, and QR 
identification. The battery is required to power 
everything in this project from Arduino to motors. 


The system software includes Arduino IDE, and 
Python IDE (Pycharm). 


Pyqt5 designer (GUI toolkit) for designing a user 
interface. 


Libraries Used: Pyserial, OpenCV, Matplotlib, 
NumPy, Cvlib, Urlib, SoftwareSerial library. 


Mediapipe: MediaPipe is a framework that is used for 
applying in a machine learning pipeline, and it is an 
open-source framework of Google. The MediaPipe 
framework is_ beneficial for cross-platform 
development since the framework is made using 
statistical data. The MediaPipe framework is 
multimodal, where this framework is often applied to 
varied audios and videos. 


Methodology: 

A. Hand gesture recognition 

The proposed system presented in this paper performs 
real-time hand gesture recognition which is done 
using various image processing techniques. The 
overall system architecture of the hand gesture 
recognition system using media pipe is shown in fig 
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B. System Architecture 
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In the block diagram, we can observe the serial 
communication between Python and Arduino Uno. 
Here Esp32 camera is working Independently, the 
live stream can be accessed by any device available if 
an IP address is known. 


Before running the Python code, First, enable the 
Bluetooth application and pair the Bluetooth HCO5 
device with a PC or laptop. When we run the python 
program it first establishes a Serial connection 
between PC to Arduino Uno. It first checks for the 
available connection, after paring it starts capturing 
video from the Webcam for live interaction between 
user and Robot. 


Then it detects for HandLandmarks of the hand and 
when it is detected successfully it counts the number 
of fingers raised and Displays it accordingly. If two 
fingers are raised it will show 2 counts. Depending on 
the number of fingers counted it will receive certain 
instructions to send commands like to go forward, 
backward, left, right, and stop. 


Later Arduino Uno receives the commands through 
serial communication between PC and HCO05 
Bluetooth. In Arduino Uno, various libraries are 
included like a Software serial that reads the received 
data. Here we first initialized the pins of Arduino Uno 
and then Begin Serial communication. It reads the 
instruction that is received and Begins commands like 
the Forward, Backward, Left, Right, and Stop. 


Result 
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The esp32 camera can be accessed remotely using its 
IP address for monitoring the robot's surroundings. 
This can be used for surveillance, monitoring, and 
other wide applications. 


The hand gesture recognition process captures the 
hand gestures of the user. The video of the user is 
captured through the webcam of a laptop or 
computer. This process involves converting the video 
to static frames which can be used for image 
processing. These frames are obtained through the 
function of the OpenCV library 
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Output: 


The movement of the robot is controlled by these five 
gestures. The captured video goes through image 
processing to detect Multi-hand landmarks. When 
landmarks are detected it counts for the number of 
figures that are raised. 


When 5 fingers are raised it sends a forward 
command. When 4 fingers are raised it sends a 
backward command. When 2 fingers are raised it 
sends the Right command. When | finger is raised it 
sends the left command. When 0 fingers are raised it 
sends a stop command. I 


Advantages: 

The main advantage of this system is that it does not 
require any additional hardware implementation as 
the web cameras of the laptop can be used to receive 
gestures from the user. It requires only a web camera 
on the laptop and hence can be used with any of the 
systems for gesture recognition. There is no need for 
contact with the computer physically and the user can 
communicate with the system from a few meters 
distance. The robotic systems controlled by hand 
gestures provide many useful applications in various 
fields. Real-time vision-based gestures can be used in 
gaming technology for designing interactive gaming 
applications. 


Applications: 

In recent years, HGR technology has made its way 
into various industries as advances in Deep Learning, 
Computer Vision, machine learning, Cameras, and 
sensors have made it more available, reliable, and 
accurate. The top four fields actively adopting hand 
tracking and gesture recognition are given below. 


> Automotive and Robotics Industry 

A gesture recognition solution from Sony Depth 
Sensing Solutions features a time-of-flight feature 
that measures the time it takes for a gesture to "travel" 
from the infrared sensor to the object and back. The 
Al is trained to differentiate main gestures from 
gestural noise and to work under any lighting 
conditions. The BMW 7 Series features a built-in 
HGR system that recognizes five gestures and can 
control music and incoming calls, among other 
things. Less interaction with the touchscreen makes 
the driving experience safer and more convenient. 


Machine learning methods, artificial intelligence, and 
complex algorithms are used by robotic control 
systems to perform a particular task, enabling natural 
communication of robotic systems with the 
environment and making autonomous decisions. 
Research studies suggest that combining computer 
vision technology with a robot can be used to create 
assistive devices which can be used by elderly. 
Another study uses computer vision to allow a robot 
to ask a human for a proper path within a house. The 
hand gesture can be used to design a surveillance 
robot used to spy on the enemies in the areas where 
humans are restricted. The dynamic feature of the 
surveillance robot increase the range of area under 
surveillance. The small size and portability of the 
robot allows it to reach inconvenient places easily. 
Search and rescue operations can be performed using 
Hand gesture controlled robots in the areas affected 
by natural calamities. These robots play a very 
important role to perform rescue operations in the 
areas which are not safe for humans 


> Healthcare Industry 

Operating rooms and emergency room may be 
chaotic, with lots of noise machines and personnels. 
In such environments, voice commands can be less 
effective than gestures. Touchscreens are not an 
option because there’s a strict boundary between what 
is and is not sterile. But access to information and 
images during surgery or another manipulation is 
possible with HGR tech. Various companies have 
proved it and many more research is ongoing on this 
system. Gesture can provide doctors with the ability 
to check Imagery, MRI and CT scans with simple 
gestures while performing their task. Patient 
monitoring in the hospitals can be done by using 
robots which can be moved from one ward to another 
using hand gestures. The robot will perform 
surveillance and keep an eye on the patients wherever 
they are present. 


> Virtual reality 
Leap Motion (acquired by Ultrahaptics) presented 
updated HGR software that allows users to track 
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gestures in virtual reality in addition to controlling a 
PC. The Leap Motion controller is a USB device that 
observes the area of about one meter with the help of 
two IR camera sensors and three infrared LEDs. A 
hand tracking application developed by ManoMotion 
recognizes gestures in three dimensions using just a 
smartphone camera (on both Android and iOS) and 
can be applied in VR and AR environments. The use 
cases for this technology include IoT devices, VR- 
gaming and interaction with virtual simulations. 


> Consumer electronics 

Home automation is another vast field within the 
consumer electronics domain in which gesture 
recognition can be used for various applications. 
uSens has developed a hardware and software to 
make smart TVs sense finger movements and hand 
gestures. Gestoos is a AI Platform in which gestures 
can be created and assigned by a smartphone or 
another device, and one gesture can be used to enable 
various commands. It also offers touch less control 
over lighting and audio systems 


The consumer market is open to accept and try new 
HMI technologies, and hand gesture recognition 
technology can be said as evolution from 
touchscreens. Demand for more smooth and means of 
interaction with devices as well as a concern for 
Safety of driver are pushing the adoption of HGR. 


Future scope: 

A limited (five) number of hand gestures have been 
considered here. Though the number of gestures can 
be extended by adding different algorithm, the 
computation will be increased. OpenCV library is 
preferred here as it is applicable for real time and 
execution is faster. It may be used on a variety of 
robotic systems with a set of gesture commands 
appropriate to that system. Further the robot can be 
designed to perform various other tasks and gestures 
can be designed accordingly. The algorithm improved 
further working in various light conditions. The 
performance analysis of the algorithm will be done by 
taking images of gestures at different conditions like 
non uniform background, poor light conditions. 


Conclusion: 

The proposed system was achieved by using a 
webcam or a built-in camera which detects the hand 
gestures. The main advantage of the system is that it 
does not require any physical contact with humans 
and provides a natural way of communicating with 
the robots. The algorithm implementation does not 
require any additional hardware. The proposed 
system can be extended further for a wide range of 
applications. Movement of robot is controlled using 
these five gestures, which were implemented. The 


arduino robot moves in the respective direction with 
the help of wheels attached to the DC motors. 
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